Mutation analysis to evaluate Lustre program specifications in the context of model-checking

Mutation analysis is a technique used to evaluate the ad- equacy of a test set with respect to a fault model. This approach is mainly used for unit testing evaluation. It is also used to produce new test data. In this paper, we apply mutation analysis in order to evaluate the accuracy of a specification in the context of reactive program verification by model-checking

Survey of source code metrics for evaluating testability of object oriented systems

Software testing is costly in terms of time and funds. Testability is a software characteristic that aims at producing systems easy to test. Several metrics have been proposed to identify the testability weaknesses. But it is sometimes difficult to be convinced that those metrics are really related with testability. This article is a critical survey of the source-code based metrics proposed in the literature for object-oriented software testability. It underlines the necessity to provide testability metrics that are proved to be intuitive and adequate for the testing cost prediction

CoMET: Compressing Microcontroller Execution Traces to Assist System Understanding

Recent technology advances have made possible the retrieval of execution traces on microcontrollers. However, even after a short execution time of the embedded program, the collected execution trace contains a huge amount of data. This is due to the cyclic nature of embedded programs. The huge amount of data makes extremely difficult and time-consuming the understanding of the program behavior. Software engineers need a way to get a quick understanding of execution traces. In this paper, we present an approach based on an improvement of the Sequitur algorithm to compress large execution traces of microcontrollers. By leveraging both cycles and repetitions present in such execution traces, our approach offers a compact and accurate compression of execution traces. This compression may be used by software engineers to understand the behavior of the system, for instance, identifying cycles that appears most often in the trace or comparing different cycles. Our evaluations give two major results. On one hand our approach gives high compression rate on microcontroller execution traces. On the other hand software engineers mostly agree that generated outputs (compressions) may help reviewing and understanding execution traces

Défis 2025

International audienceNew paradigms, languages, modeling, verification, testing approaches and new tools in the field of programming and software should be created in the next 10 years, whether to make life easier for designers and maintainers of computer systems, to model and reliable software or to anticipate technological change. This text summarizes the challenges in the Programming and Software Engineering field on the horizon 2025. This work has been presented and discussed during the national days of the Research Group on Programming and Software Engineering in June 2014 and in September 2014 in Paris.De nouveaux paradigmes, de nouveaux langages, de nouvelles approches de modélisation, de vérification, de tests et de nouveaux outils dans le domaine de la programmation et du logiciel devraient voir le jour dans les dix ans à venir, que ce soit pour faciliter la vie des concepteurs et mainteneurs de systèmes informatiques, pour modéliser et fiabiliser les logiciels ou encore pour devancer l’évolution technologique. Ce texte résume les travaux menés sur les défis du Génie de la Programmation et du Logiciel à l’horizon 2025. Ces travaux ont été l’occasion de présentations et d’échanges lors des journées nationales du Groupe de Recherche Génie de la Programmation et du Logiciel en juin 2014 et lors d’une journée en septembre 2014 à Paris

Conformance testing from UML specifications. Experience Report

International audienceUMLAUT is a framework for building tools dedicated to the manipulation of models described using the Unified Modeling Language (UML). TGV is a tool for the generation of conformance test suites for protocols. Both tools are connected so that it is possible to specify an application in UML and derive automatically some test cases. In this article, the integration of those tools in an industrial process is evaluated through a case study. This case study, proposed by Gemplus, is a Java Card applet: a classical electronic purse

CoMET: Compressing Microcontroller Execution Traces to Assist System Understanding

Les rapports de recherche du LIG - ISSN: 2105-0422Recent technology advances have made possible the retrieval of execution traces on microcontrollers. However, even after a short execution time of the embedded program, the collected execution trace contains a huge amount of data. This is due to the cyclic nature of embedded programs. The huge amount of data makes extremely difficult and time-consuming the understanding of the program behavior. Software engineers need a way to get a quick understanding of execution traces. In this paper, we present an approach based on an improvement of the Sequitur algorithm to compress large execution traces of microcontrollers. By leveraging both cycles and repetitions present in such execution traces, our approach offers a compact and accurate compression of execution traces. This compression may be used by software engineers to understand the behavior of the system, for instance, identifying cycles that appears most often in the trace or comparing different cycles. Our evaluations give two major results. On one hand our approach gives high compression rate on microcontroller execution traces. On the other hand software engineers mostly agree that generated outputs (compressions) may help reviewing and understanding execution traces

Feature Interaction Detection using Testing and Model-checking - Experience Report

. We present an experiment in feature interaction detection. We studied the 12 features defined for the first feature interaction contest held in association with the 5th international Feature Interaction Workshop. We used a synchronous approach for modeling features, and both, a model-checker and a test generator for revealing interactions. The first part of the paper describes the feature modeling. The second part deals with the feature interaction detection carried out with a testing tool, and the last part addresses the use of a model-checker for the detection. 1 Introduction Telecommunication software is a variety of safety-critical software. Its requirements in terms of dependability are high since a malfunction may result in environment harm. The disastrous financial consequences of failures impose on this kind of software strong correctness and quality of service constraints. This is why modeling, analysis and risk assessment activities take a large part of its development pro..

Test fonctionnel statistique de logiciels spécifiés en Lustre ; application à la validation de services téléphoniques

This work deals with the design of functional testing environments for reactive systems formally specified in Lustre. Lutess is such an environment. It enables test data generation automatically and dynamically. The generation is constrained, randomly performed, and possibly guided with properties. We have extended this environment with a statistical testing method, which favors the generation of test data considered to be relevant by the user. We have validated experimentally Lutess before and after its extension, showing that the test data are indeed randomly generated, and that they follow the expected statistical distribution. Moreover, we have intensively used Lutess and the proposed testing method for the validation of telephony feature specifications. This work has been done on two case studies : one provided by the CNET and another one, a contest, proposed in conjunction with "the fifth Feature Interaction Workshop" (1998). On that occasion, Lutess won the "best feature interaction detection tool award".Ce travail s'inscrit dans le cadre de la conception d'environnements de test fonctionnel de systèmes réactifs spécifiés formellement en Lustre. Lutess est un tel environnement. Il permet la génération automatique et dynamique de données de test. Cette génération est menée sous contraintes, de manière aléatoire, et éventuellement guidée par des propriétés. Nous avons étendu cet environnement par une méthode de test de type statistique, qui facilite la génération des données de test considérées comme significatives par l'utilisateur. Nous avons validé expérimentalement Lutess avant et après son extension, en montrant que les données générées étaient bien aléatoires et qu'elles respectaient les distributions statistiques attendues. De plus, nous avons utilisé Lutess et la méthode proposée de façon intensive pour la validation de spécifications de services téléphoniques sur deux études de cas conséquentes : une fournie par le CNET et une autre sous la forme d'un concours proposé en marge de la conférence "Feature Interaction Workshop'98". A cette occasion, Lutess a été déclaré "meilleur outil pour la détection d'interactions de services téléphoniques"

Contributions à l'automatisation raisonnée de différents processus du test logiciel

Testing is the main activity in software validation. For reducing the testing costs and increasing the system under test quality, it is essential to provide automated test solutions and to ensure the productivity of test engineers. The work presented in this document aims at contributing to these objectives. These works are based on three axes. The first one concerns test generation. The originality of the work relies in establishing principles for the production of combinatorial test suites from abstract expressions. These principles have been implemented in a tool called Tobias. When large test suites are produced, it is important to automate the test oracle. A second focus is to assess the usability of assertions to determine the test verdicts for domotic applications. The results show that the assertions can actually be used, but that the execution of these applications in a live environment, which is not completely controllable and observable, may result in biaised verdicts. Thus, automating test generation and test oracle reduces the cost of the testing phase. But to reduce the cost of testing, it is also important to consider the internal factors related to the software testability. Many metrics have been proposed to predict and estimate the testability of a system. A third line of research concerns the evaluation and the experimental validation of these metrics. The results of experiments demonstrate that the studied metrics cannot be directly used to predict the test cost.Le test constitue aujourd'hui la principale activité de validation d'un logiciel. Dans un contexte où l'on cherche à réduire les coûts et augmenter la qualité, il est essentiel de proposer des solutions de test automatisées et de veiller à la productivité des ingénieurs de test. Les travaux présentés dans ce mémoire ont l'ambition de contribuer à ces objectifs. Ces travaux se déclinent selon trois axes. Le premier axe concerne la génération de tests. L'originalité du travail se situe dans l'établissement de principes pour la production combinatoire de suites de test, à partir d'expressions abstraites. Ces principes ont été implantés dans un outil appelé Tobias. Lorsque de grandes suites de test sont produites, il est important d'automatiser l'oracle des tests. Un deuxième axe vise à évaluer l'utililisabilité des assertions pour établir le verdict des tests pour des applications domotiques. Les résultats montrent que les assertions sont effectivement utilisables, mais que l'exécution de ces applications dans un environnement réel non complètement contrôlable ou observable peut conduire en l'émission de verdicts biaisés. Ainsi, l'automatisation de la génération et de l'oracle permet de réduire le coût de la phase de test. Mais, pour réduire le coût du test, il est aussi important de considérer les facteurs internes au logiciel relatifs à la testabilité. De nombreuses métriques ont été proposées pour prédire et estimer la testabilité d'un système. Un troisième axe de recherche concerne l'évaluation la validation expérimentale de ces métriques. Les résultats des expérimentations démontrent que les métriques étudiées ne sont pas directement utilisables pour prédire le coût du test

Directed random reduction of combinatorial test suites. of ACM for your personal use. Not for redistribution. The definitive version was published in Proceeding RT '07 Proceedings of the 2nd international workshop on Random testing: co-located with the 22

Abstract Combinatorial testing consists in generating (possibly large) test suites by combining both the sequencing of several operations, and the selection of test data. The TOBIAS tool is based on this generation technique. The combinatorial part of the approach makes both its strength and its weakness. Indeed, the more tests are produced, the more confidence the user may have in his/her program. Nevertheless, simple patterns may result in millions of test cases due to the combinatorial explosion problem, leading to intractable testsuites. To overcome this weakness, TOBIAS makes it possible to connect the generator to ``selectors'' that choose a subset of the test suite based on various techniques or criteria. This paper presents such a test suite reduction technique, based on a stochastic approach. The selection is improved so a uniform repartition of test cases is performed w.r.t. a given criterion, namely, an abstraction of the test cases based on their ``shape''. The approach is implemented into TOBIAS as a generic selector, independent of the system under test. It has been put into practice on a smart card case study, for which the results are reported in this paper. This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder. ABSTRACT Combinatorial testing consists in generating (possibly large) test suites by combining both the sequencing of several operations, and the selection of test data. The TOBIAS tool is based on this generation technique. The combinatorial part of the approach makes both its strength and its weakness. Indeed, the more tests are produced, the more confidence the user may have in his/her program. Nevertheless, simple patterns may result in millions of test cases due to the combinatorial explosion problem, leading to intractable testsuites. To overcome this weakness, TOBIAS makes it possible to connect the generator to "selectors" that choose a subset of the test suite based on various techniques or criteria. This paper presents such a test suite reduction technique, based on a stochastic approach. The selection is improved so a uniform repartition of test cases is performed w.r.t. a given criterion, namely, an abstraction of the test cases based on their "shape". The approach is implemented into TOBIAS as a generic selector, independent of the system under test. It has been put into practice on a smart card case study, for which the results are reported in this paper